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Abstract 

This paper examines the most probable route to chaos in high-dimensional dy- 
namical systems in a very general computational setting. The most probable route 
to chaos in high-dimensional, discrete-time maps is observed to be a sequence of 
Neimark-Sacker bifurcations into chaos. A means for determining and understand- 
ing the degree to which the Landau-Hopf route to turbulence is non-generic in 
the space of mappings is outlined. The results comment on previous results of 
Newhouse, Ruelle, Takens, Broer, Chenciner, and looss. 
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In their first edition of Fluid Mechanics [26], Landau and Lifschitz proposed 
a route to turbulence in fluid systems. Since then, much work, in dynamical 
systems, experimental fluid dynamics, and many other fields has been done 
concerning the routes to turbulence. In this paper, we present early results 
from the first large statistical study of the route to chaos in a very general 
class of high-dimensional, C^, dynamical systems. Our results contain both 
some reassurances based on a wealth of previous results and some surprises. 
We conclude that, for high- dimensional discrete-time maps, the most probable 
route to chaos (in our general construction) from a fixed point is via at least 
one Neimark-Sacker bifurcation, followed by persistent zero Lyapunov expo- 
nents, and finally a bifurcation into chaos. We observe both the Ruelle- Takens 
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scheme as well as persistent n-tori, where n < 2 before the onset of chaos. 
The point of this report is two-fold. First, we propose a manageable set of 
functions that will yield a better general understanding of high-dimensional 
routes to chaos in a practical sense. Second we provide a survey of the routes 
to chaos in this proposed set of mappings. The goal is to select a common set 
of mappings that exists someplace between real-world examples (the set we 
propose can approximate a very general set of real world examples), and the 
low-dimensional cases considered in rigorous mathematics. 

Begin with an ordinary differential equation in with a single real parameter 
fj,, ^ = F{ii, U) where F is as smooth as we wish and J7 C i?^ is compact. 
At fiQ there exists a fixed point, and at fic, fJ'O < l^c, F is chaotic. The bifur- 
cation sequence proposed by Landau [26] and Hopf [20] is the following: as 
II is increased from /iq there will exist a bifurcation cascade of quasi-periodic 
solutions existing on higher and higher dimensional tori until the onset of 
"turbulence." In other words, the solutions would be of the following type, 

lii < i^i+i, and where none of the frequencies are rationally related. However, 
Landau and Hopf 's notion of turbulence was high-dimensional, quasi-periodic 
flow. Ruelle and Takens [33] proposed both an alternative notion of turbu- 
lence (the strange attractor) and an alternative route to turbulence in a now 
famous paper. Ruelle and Takens claimed that the Landau path was highly 
unlikely from a topological prospective. The basis for their claim originates in 
the work of Peixoto [30] who has shown that quasi-periodic motion on (the 
2-torus) is non-generic^ in the set of C" vector fields. However, Peixoto's the- 
orem applies only to fiows on and not T'^ for k > 2. For diffeomorphisms 
of the circle, irrational rotations make up the full Lebesgue measure set of 
rotations. Suspensions of such diffeomorphisms correspond to quasi-periodic 
motion on T^. It would seem that quasi-periodic motion of flows on would 
be high measure, however, there is not a one-to-one correspondence between 
fiows on and discrete-time maps of the circle (e.g. the Reebs foliation or 
[29]). Further, quasi-periodic orbits of diffeomorphisms on the circle are struc- 
turally unstable. For fiows on T^, the structurally stable, hyperbolic periodic 
orbits are topologically generic; however, it is likely that quasi-periodic orbits 
are common in a measure theoretic sense on T^. There remain many open 
questions regarding bifurcations of periodic orbits, for example, a list of codi- 
mension 2 bifurcations and their status can be found at the end of section 
(9.1), page 397 of [25]. How this will all play out in practice is unclear and 
comprises a good portion of motivation for our study. Ruelle and Takens then 
went on to prove two results for fiows relevant to this report. The first is a 
normal form theorem for the "second" Hopf bifurcation for vector fields, or 
the "first" Hopf bifurcation for maps (often referred to as the Neimark-Sacker 



A property is generic if it exists on subset E <Z B, where E contains a countable 
intersection of open sets that are dense in the original set B. 



2 



bifurcation [34] [27]). This theorem gives a normal form analysis of the bi- 
furcation of an invariant circle of a flow, but it does not state the type of 
dynamic that will exist upon the loss of stability of the invariant circle. The 
second relevant result was that, given a quasi-periodic solution /(tui, . . . ,Uk) 
on T'^, A; > 4, in every C''~^ small neighborhood of (a;i, . . . ,uJk), there exists 
an open set of vector fields with a strange attractor (prop. 9.2 [33]). These 
results were extended by Newhouse, Ruelle and Takens [28] who proved that a 
perturbation of a quasi-periodic flow on can produce strange (axiom A) 
attractors, thus reducing the dimension to three for tori with quasi-periodic 
solutions for which an open set of perturbations yield strange attractors. 
The basic scheme used by Takens, Ruelle and Newhouse was to first prove a 
normal form theorem from periodic orbits to 2-tori in vector fields and then 
prove something about how the 2-tori behave under perturbations — showing 
that bifurcations of m-tori to (m -|- l)-tori will yield chaos since 3-tori can be 
perturbed away to axiom A chaotic attractors. 

If the story were only as simple as a disagreement between topological and 
measure theoretic viewpoints, we would be in good shape. However, out of the 
complexity of the dynamics and the difficulties posed by bifurcation theory 
regarding what happens to bifurcations of resonant periodic orbits and quasi- 
periodic orbits, the field of quasi-periodic bifurcation theory was born [10] [7]. 
We will not attempt to discuss the history from where the above leaves off to 
the current state. For those interested, see [8], [12], [23], [13], [9], [24], and [22]. 
The question regarding the most common route to turbulence is, in any but a 
very select set of specific examples, still an open and poorly defined question. 
Analytically piecing together even what types of bifurcations exist en route 
to chaos has been slow and difficult. We contribute to the existing partial 
solution in the following ways: we will provide a framework for numerical 
analysis that does not have a priori tori built in; we will provide evidence 
that the dominant scenario with respect to the cascade of bifurcations leading 
to chaos in many high- dimensional dynamical systems will consist of various 
bifurcations between periodic orbits of high-period and quasi-periodic orbits. 
Moreover, fiip and fold bifurcations (due to real eigenvalues) occur with less 
and less frequency as the dimension of the dynamical system is increased. 

Let us begin by arguing why we believe that, as the dimension of the dynami- 
cal system is increased, fiip, fold, any bifurcations due to real eigenvalues, and 
strong resonance bifurcations will be vanishingly rare and the path to chaos 
from a fixed point in parameter space will consist of a soup of various peri- 
odic orbits with high-period and quasi-periodic orbits. Our intuition is born 
from results in the random matrix theory of Girko [17], Edelman [14], and 
Bai [4]. The basic idea of their work is the following: given a square matrix 
whose elements are real random variables drawn from a distribution with a 
finite sixth moment, in the limit of infinite dimensions, the normalized spec- 
trum (or eigenvalues) of the matrix will converge to a uniform distribution on 
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the unit disk in the complex plane. It is worth noting that the convergence 
in measure is not absolutely continuous with respect to Lebesgue measure. 
Nevertheless, if the Jacobian of the map at the "first" bifurcation point (i.e. 
the bifurcation from the fixed point) is a high-dimensional matrix whose ele- 
ments have a finite sixth moment, it is reasonable that the bifurcation would 
be of type Neimark-Sacker (via a complex eigenvalue) instead of a flip or fold, 
with probability approaching unity as the dimension goes to inflnity. Following 
this line of thought, we, with W. Davis Dechert, provided strong numerical 
evidence that the first bifurcation in typical high-dimensional discrete-time 
dynamical systems would be that of the Neimark-Sacker type [3] . Extending 
this analytically for the fc*'* bifurcation before the onset of chaos where k > 1 
is made difficult (a proof for the k = 1 case is not difficult to construct) by 
the fact that the normal form and center manifold theory for quasi-periodic 
bifurcation theory is a long way from providing a general form(s) about which 
the results of the random matrix theory could be applied (the codimension 
2 situation is not complete yet, and the codimension 3 case is even further 
from completion). However, it is assumed that, in the end, most bifurcations 
of periodic and quasi-periodic orbits can be captured by some sort of Taylor 
series expansion (via a vector field approximation or a suspension). Though 
the linear term of the Taylor expansion will be degenerate and the outcome 
of the bifurcation will be determined by contributions of higher-order terms, 
the degeneracies in the linear term of the Taylor polynomial will nevertheless 
be of complex eigenvalues — leading to some sort of a bifurcation (yet to be 
understood) from quasi-periodic or periodic orbits to other quasi-periodic or 
periodic orbits. 

A full statistical study is hampered by numerical accuracy issues which may 
be fundamental problems associated with the existence of neutral directions. 
We will present the most clear picture of the prototypical route to chaos in 
high dimensions for our set of functions via a prototypical example. For the 
understanding reached in this report, we will employ bifurcation diagrams, 
phase plots, the largest Lyapunov exponent and the full Lyapunov spectrum. 
Numerical issues related to the Lyapunov spectrum are the main problem with 
respect to the statistical study, and we will discuss and display such problems. 
For information regarding why the numerical problems (such as truly infinite 
convergence times) we encounter may be unavoidable, see [31] and [11] for the 
latest with regard to the existence of zero Lyapunov exponents and [18] [19] 
with respect to the linear algebra issues with computing them. At the end 
of the report we will note the numerous extensions our current line of study 
suggests. 

The example we will present is typical (we will discuss what is not typical 
about our example during the analysis) amongst the 500 or so cases (with this 
number of parameters and dimensions) we have observed in the sense that 
between the first bifurcation and the onset of chaos, the only type of orbit 
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that exists is either quasi-periodic or periodic orbits with periods high enough 
such that they are indiscernible from quasi-periodic orbits. It is worth noting 
that the route to chaos we discuss in this report, which we beheve is the typical 
route in high dimensions, is considerably different from what is observed at 
low dimensions. For an intuitive feel for the lower-dimensional cases, see [3]. 

A primary tool of analysis will be the Lyapunov exponent spectrum ([6] [32]), 
since it is a good measure of the tangent space of the mapping along its 
orbit. Negative Lyapunov exponents correspond to global stable manifolds 
or contracting directions; positive Lyapunov exponents correspond to global 
unstable manifolds or expanding directions and are ([32]), in a computational 
framework, the hallmark of chaos. A zero exponent corresponds to a neutral 
direction (although the story in this case is significantly more complicated, 
see [11]); when the largest Lyapunov exponent for a discrete-time map is 
zero and all other exponents are less than zero, then there exists a neutral 
direction. Lyapunov exponents relate to quasi-periodic orbits and rotations in 
the following way, neutral rotating directions correspond to a zero Lyapunov 
exponent, and high-period orbits correspond to pairs of negative Lyapunov 
exponents. If the largest exponent is zero (while all others are negative), then 
there exists a quasi-periodic orbit on the circle (on in the flow). If two 
exponents are zero, then there is a full 2-torus in discrete-time (a 3-torus in 
the flow), and so on (for speciflc examples of high- dimensional tori see [9]). 
With respect to the codimcnsion 2 bifurcations, there can also be as many as 10 
possibilities for zero Lyapunov exponents at bifurcation points (again see page 
397 of [25] ) , some of which correspond to real and complex conjugate pairs of 
eigenvalues. Besides the full Lyapunov spectrum algorithm, we will also use 
an independent version of a calculation of the largest Lyapunov exponent a la 
Wolf et. al [38]. Upon analysis of the example, it will be obvious why we need 
such a calculation. 

The discrete-time mappings we will consider are single-layer, feed-forward neu- 
ral networks of the form 



which is a map from to i?. In cq.(l), n represents the number of hidden units 
or neurons, d is the input or embedding dimension of the system which func- 
tions simply as the number of time lags, and s is a scaling factor on the weights. 
The parameters are chosen in the following way: Wij, Xj,s e R, where the 
/Sj's and Wij^s are elements of weight matrices (which we hold flxed for each 
case), (xq, Xi, . . . , Xd) represent initial conditions, and (x^, Xt+i, . . . , Xt+d) rep- 
resent the current state of the system at time t. We assume that the /9's are 
iid uniform over [0,1] and then re-scaled to satisfy YH=il3i = n. The Wi/s 
are iid normal with zero mean and unit variance. The s parameter is a real 
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Fig. 1. Bifurcation diagram for a typical network; n = 32, d = 64. 

number and can be interpreted as the standard deviation of the w matrix of 
weights. The initial chosen iid uniform on the interval [—1, 1]. 

We would like to make a few notes with respect to the squashing function, 
tanh(). For x very near 0, the tanh(a;) function is nearly linear. Moreover, 
tanh(x), for |x| ^ 1, will tend to behave much like a binary function (±1). 
Thus the scaling parameter s will provide a unique bifurcation parameter 
that will sweep from linear ranges to highly non-linear ranges, to nearly binary 
ranges — fixed points to chaos to what seem like periodic cycles. In this report 
however, we will cease consideration of our networks long before the neurons 
become saturated. 

Scalar neural networks like these arc "universal approximators," meaning they 
can approximate many very general spaces of mappings (e.g. any mapping 
and its derivatives to arbitrary accuracy, given enough neurons). That scalar 
neural networks can approximate the mappings we are interested in is a topic 
addressed in [21]. Combining the approximation theorems of Hornik ct al [21] 
and the time-series embedding results of Takens [36] shows the equivalence 
between our neural networks and the dynamical systems from compact sets 
in i?" to themselves (for specific arguments along these lines, see [1]). 

To circumvent the numerical stability issues that will be apparent shortly, we 
are required to consider each network individually with four types of figures: 
the largest Lyapunov exponent independently computed of the full spectrum 
(it is considerably more numerically stable than the algorithm for the full spec- 
trum); a standard bifurcation diagram; the full Lyapunov spectrum computed 
in the standard manner; and phase-space diagrams. 

Our choice of the number of neurons and the number of dimensions is based 
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Fig. 2. The Largest Lyapunov for a typical network; n = 32, d = 64. 

on Fig. (1) of [3] and the compromise required by computational time hmits. 
Considering Fig. (1) of [3], 32 neurons puts our networks deeply in the region of 
the set of neural networks that correspond to highly complicated and chaotic 
dynamics. The dimension, 64 was chosen because it was the highest dimension 
for which we could reliably compute the nearly 500 cases we surveyed. The 
compute time increases as a power of the dimension. Thus as this time, we 
had too few cases of d = 128 and d = 256 to make conclusive statements. 

Beginning with Fig. (1), the standard bifurcation diagram, there are four 
important features to notice. The first feature is the first bifurcation, which 
occurs at s = 0.0135 from a fixed point to some type of limit cycle or torus. 
A secondary bifurcation is clearly visible at s = 0.02755, the nature of this 
bifurcation is entirely a mystery from the perspective of Fig. (1). Chaos seems 
to onset near s — 0.05, and has definitely onset by s = 0.06, however the exact 
location is difficult to discern. Lastly, all of the dynamics between the fixed 
point and chaos are some sort of n-torus {n > 0) type behavior. 

Next, let us consider Fig. (2) — the largest Lyapunov exponent versus vari- 
ation in s for the same case as Fig. (1). Again, as expected, we see the first 
bifurcation at s = 0.0135, in agreement with the bifurcation diagram. Fig- 
ure (2), however, gives a clear picture of the onset of chaos, which occurs at 
s = 0.05284. The largest exponent is near zero between the first bifurcation 
and the onset of chaos, providing evidence for the existence of at least one 
persistent complex conjugate pair of eigenvalues with modulus one (assuming 
a Jacobian can be constructed) — e.g. a persistent quasi-periodic orbit. 

Considering Fig. (2), near the onset of chaos the exponent becomes negative 
over a very short s interval. Ignoring all the intermediate bifurcations, let us 
briefly consider the onset of chaos via Fig. (3). Considering this figure, there 
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Fig. 3. The Largest Lyapunov at the onset of chaos. 
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Fig. 4. Phase plots on either side if the 2"°' bifurcation, s = 0.0275 and s = 0.029 
respectively. The bifurcation occurs at s ~ 0.02754. 

is an apparent periodic orbit, followed but what might be a period doubling 
bifurcation, followed by what could be a complicated bifurcation structure. 
Besides noting this for general interest and completeness, we will refrain from 
a further discussion of this small interval since this behavior seems to disappear 
for high- dimensional networks and is not particularly related to the point of 
this report. 

We will begin our presentation of phase-space figures at the second bifurcation. 
The second bifurcation is the obvious in the bifurcation diagram, correspond- 
ing to the rapid change in the attractor size near s ~ 0.0275. The locations 
and nature of these bifurcations can't be determined by a consideration of the 
Lyapunov spectrum or the largest Lyapunov exponent, although considering 
the Lyapunov spectrum it is clear that such a bifurcation does occur. The 
second bifurcation appears to be from a 1-torus to a 2-torus as shown in Fig. 
(4). 



The phase-space plots on either side of the third bifurcation, which is not 
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Fig. 6. Phase plots on either side of the 4 bifurcation, s = 0.0512 and s = 0.0514 
respectively. The bifurcation occurs at s ~ 0.05124 

clearly apparent in the bifurcation diagram or in the Lyapunov spectrum as 
we will see, is depicted in Fig. (5). By this point, the smooth looking torus- 
like object from Fig. (4) has become "kinked" significantly before the third 

bifurcation. At the third bifurcation, the torus-like, two-dimensional object 
becomes a onc-dimensional object. Thus the third bifurcation is from a 2- 
torus to a 1-torus and occurs at s ~ 0.0466. The 1-torus is a severely "kinked" 
quasi-periodic orbit. 

Considering the phase-space plots on either side of the fourth bifurcation in 
Fig. (6), one might conclude that there our example has undergone a period 
doubling of the quasi-periodic orbit. An analytical explanation of such a bi- 
furcation is yet an open problem, but it is hkely a "Neimark-Sacker-Flip" 
bifurcation. Wc will refrain from a further discussion of this bifurcation, di- 
recting the interested reader to chapter 9 of [25]. 

We will refrain from showing figures for the fifth bifurcation, simply noting 
that it is a bifurcation from this quasi-periodic orbit on the 1-torus to a high- 
period, periodic orbit. Rather, we will skip to what we will call the sixth 
bifurcation. Figure (7) demonstrates the final bifurcation into chaos from a 
high-period, periodic orbit. However, in our particular example, considering 
Fig. (3), just before the onset of chaos, there is a likely a sequence of bifurca- 
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Fig. 7. Phase space plot near the 6*^ bifurcation — s = 0.052 and s = 0.053 
respectively. The bifurcation occurs at s ~ 0.05294. 




Fig. 8. Lyapunov spectrum the (typical) network; n = 32, d = 64. 

tions. We will not belabor this further, as little insight is gained from a further 
consideration, and this sequence of bifurcations just before the onset of chaos 
appears to be increasingly rare as the dimension increases. 

Before discussing what can be gleaned from the Lyapunov spectrum data, we 
must make a few comments with respect to the data plotted in Fig. (8). If one 
were to pick an s value along the s interval (0, 0.03) and count the number 
of Lyapunov exponents, it is likely that one could often only identify roughly 
half the number of exponents that might be expected. There is a simple reason 
for this, if one were to consider the eigenvalues of the variational differential 
equation, ^ nearly all the eigenvalues are complex conjugate pairs, and hence 
the pairs of nearly equal Lyapunov exponents. This in a sense, implies a lot of 
local rotation, but as this can depend upon the coordinate system, little can be 

^ See page 36 of [5] for a definition of the variational differential equation. In prac- 
tical terms, we are referring to something akin to the eigenvalues of the local "Ja- 
cobian" of the network at a specific time. 
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said in this regard. Nevertheless, from a practical point of view, this implies 
sink behavior in all directions. The point is that the "largest" exponent is 
actually pair of nearly equal exponents. This is true of at least the ten largest 
"exponents," and often this is true of all of the "exponents" on (0,0.03). The 
Neimark-Sacker style bifurcations to periodic orbits and tori are due to those 
pairs of exponents. If the bifurcations were of real eigenvalues, there would 
not be pairs of Lyapunov exponents of equal magnitude. 

On the s interval ~ (0,0.03), nearly all the exponents come in pairs, all the 
lines represent two exponents. Much of this is related to the numerical stability 
issues related to the linear algebra engines used to calculate the exponents. 
This is a fundamental and common problem in matrices with many nearly 
equal, and hence degenerate eigenvalues. This issue lies in the standard algo- 
rithm's ability to accurately orthogonalize the eigenvalues and eigenvectors, 
for more information see [18] and [19]. At s ~ 0.03 the negative exponents 
begin to split apart. This cascade continues to occur from the most negative 
exponents to the largest exponents, in that order, up to the onset of chaos. By 
the onset of chaos, there exists 64 distinct exponents, one for each dimension. 
In fact, by s ~ 0.052, all the exponent pairs are observably spht. It is possi- 
ble that this is the first time this phenomena has been observed. There are 
several interpretations for such behavior; we will sketch our interpretation. In 
the interval where the exponents exist in nearby pairs, the dynamics are dom- 
inated by rotation in nearly all eigen-directions, and the many nearly equal 
local eigenvalues cause numerical stability problems. 

As the s parameter is increased, the, rotating directions become less degener- 
ate, and the lower exponents begin to separate. This phenomena has very little 
to do with the overall dynamics when the eigenvalues are strongly negative 
since these directions are so strongly contracting that they are not observed on 
the attr actor after transients die out. As the s parameter is further increased, 
more pairs of Lyapunov exponents separate until the neutral directions begin 
to separate — this is the break up of the tori and the onset of turbulence. ^ 
We are displaying an example of the Ruelle-Takens scenario which contrasts 
the Landau and Hopf turbulence model. Strange attractors are not a collec- 
tion of interacting quasi-periodic orbits or a rotating soup, but rather, distinct 
directions of expansion, contraction a la axiom A, and a little bit of rotation 
(neutral directions). This example, and Fig. (8) display this distinction in a 
very nice way. This cascade from all commensurate to all incommensurate 
Lyapunov exponents was unexpected. 

The first bifurcation is relatively easy to identify in Fig. (8), it occurs at a 
relative maximum of the largest overlaying set of exponents in the spectrum 



^ With the break up of the commensurate directions, our numerical stabiUty re- 
turns, see [2] or [1] for more regarding the dynamics after this transition 
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at s = 0.0135. This signals the Neimark-Sackcr type first bifurcation, as ex- 
pected. In our experience, the first bifurcation can be determined from the 
full spectrum reliably. The second bifurcation is also, relatively obvious at 
s — 0.02755, and corresponds to the relative maximum of the six (three pairs) 
largest exponents. This would signal the existence of a torus of some type, but 
due to the numerical stability issues it is difficult to make a definitive state- 
ment regarding the dynamics. Nevertheless, by the second bifurcation point, 
there clearly exists at least a 2-torus. Near the third bifurcation the Lyapunov 
spectrum becomes significantly more complicated — due to the numerical ac- 
curacy issues we dare not make a claim with respect to how many exponents 
are actually zero; however, it appears as if there are several. The onset of chaos 
is vague, but chaos can clearly be identified by s = 0.8. Likewise, considering 
Figs. (8) and (2), the respective largest Lyapunov exponents begin to be in- 
distinguishable at s = 0.08. The secondary bifurcations cannot be identified, 
however, and it is this problem that has hindered the full statistical study. 

The previous information can be summarized in the following table for which 
the location of the various bifurcations is given as observed via the bifurcation 
diagram, the largest Lyapunov exponent, the phase space diagrams, and the 
Lyapunov spectrum: 
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Putting this all together into a unified picture, it is likely that the first bifur- 
cation, of type Neimark-Sacker, occurs at s = 0.0135. The second bifurcation 
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occurs at s = 0.02754 is a bifurcation from to T^. The third bifurcation, 
which occurs at s = 0.04667 is a bifurcation from back to T^. The fourth 
bifurcation appears to be a period-doubhng Uke bifurcation from to T^. The 
fifth bifurcation is a bifurcation from the quasi-periodic orbit to a very high- 
period orbit. Following this is a very subtle sequence of bifurcations followed 
by the onset of chaos at s = 0.05289. 

All the other observed cases with d = 64 were variations on a theme. We have 
observed more and fewer bifurcations between the first bifurcation and chaos, 
but all the bifurcations are relatively similar. Period-doubling cascades and 
such routes are rare — they occur in less than one percent of 64-dimcnsional 
networks with our weight distribution (i.e. we did not observe a period dou- 
bling route to chaos in the 500-1- networks we considered for this report). Only 
five percent of the first bifurcations are due to real eigenvalues, and there never 
exist cascades of multiple real bifurcations. The most common route to chaos 
we observed is a cascade of bifurcations between T^, T^, and high-period, pe- 
riodic orbits; 3-tori are rare in our experience, we never observed one in nearly 
500 cases. 

One strength of our methods and results is, unhke topologically based results, 
our neural network framework allows not only a practical means of analyzing 
topological results, but it contains, in a manageable way, the supposed patho- 
logical examples. Due to the mappings that neural networks can approximate, 
if the spectrum of Lyapunov exponents of a d-dimensional network is con- 
tained in [xi, Xd], then likely there exists at least one path through parameter 
space such that any network can be transformed into a T'^ torus with all Lya- 
punov exponents being zero (we haven't proved such a result, however). If 
one were to stratify the networks by their spectra, the aforementioned torus 
would be but a point along the interval [xi,Xd]; and thus, in this sense, un- 
usual. There are other stratifications of the networks that can be made, such 
as the map (j) : R'^(^+^) — > E(tanh()), where E(tanh()) is the set of neural net- 
works with tanh() as the squashing function. The point is, we are presenting 
a practical framework that yields numerical observations regarding common 
routes to chaos. For example, the number of constraints required to achieve a 
(i-torus might be high ^ , and the torus would hkely exist on a surface in 
parameter space of much lower dimension than the ambient parameter space. 
However, since the neural networks can approximate m-tori for m < d, it is 
possible to study the transitions, and the likelihood of such transitions, and the 
persistence of high-dimensional tori in a practical way. From our experience 
perturbing away 2-tori and limit cycles requires drastic parameter variation, 
however, we never observe 3-tori consistent with the prediction of Newhouse, 



^ The number of constraints for a d-torus might also be low, as requiring area 
preservation, or a local Jacobian to have an average determinant equal to ±1 is, in 
a sense, a singe constraint. 
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Ruelle and Takens. 

As this paper is intended as a survey of our space of neural networks along 
the route to chaos, let us list current and future directions of work: 

i. a specific numerical analysis of each of the bifurcations in this example that 
can be compared to the theory; specifically the second and third bifurcations 
which are likely of low codimension; 

ii. an analytical normal form calculation — again starting with the early bi- 
furcations along the route; 

iii. a study of Lyapunov spectrum calculation technique a la [16], [15], [37], 
[6] , or [35] — these networks form a nice set of high-dimensional mappings 
to study Lyapunov spectrum calculation schemes since our mappings are 
high dimensional, not pathological, and are not, as high-dimensional maps 
go, computationally intensive to use; very often, upon the presentation of 
a new Lyapunov spectrum computation algorithm the test cases are very 
low dimensional dynamical systems for which numerical stability is rarely a 
problem; 

iv. a brute force — by hand — statistical study of the bifurcation sequences in 
these networks; 

V. a more numerically accurate Lyapunov spectrum calculation routine that 
can be used for a full statistical analysis of the routes to chaos in this set of 
dynamical systems; 

vi. a systematic investigation of the dependence on the number of neurons; 

vi. a systematic investigation of the sensitivity of the weight distribution; 

vii. constraints on weight distributions which lead to dynamics similar to a 
particular physical phenomenon, leading to a better understanding for how 
special a physical phenomena is relative to a general function space. 

Our results help extend the current analytical results in the sense that we 
have a practical way of observing transitions along an interval in parame- 
ter space. Further, we present a framework such that the parameter set for 
which the claimed non-generic tori and persistent quasi-periodic behavior can 
be more concretely understood. Based on our observations, in high dimen- 
sions, the quasi-periodic route to chaos, often with a cascade of bifurcations is 
the dominant route. We observe what is predicted to occur according to the 
Rucllc-Takcns route, but the bifurcation cascade is significantly more compli- 
cated than ^ ^ T"^ ^ chaos. The bifurcation cascade often involves 
successive bifurcations between tori with dimension< 2. 

We would hke to thank R. A. Bayliss and W. D. Dechert for many helpful 
discussions and advice. D. J. Albers would like to give special thanks to J. 
P. Crutchfield for guidance, many fruitful discussions, wonderful insight and 
feedback, and support. The computing for the project was done on the Be- 
owulf cluster at the Santa Fe Institute and was partially supported at the 
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tel Corporation under the Computation, Dynamics, and Inference Program 
via SFI's core grant from the National Science and MacArthur Foundations. 
Direct support for D. J. Albers was provided by NSF grants DMR-9820816 
and PHY-9910217 and DARPA Agreement F30602-00-2-0583. 
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